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1. INTRODUCTION 

The optimization problem finds application in several fields, such as classical continuum physics, 
theoretical, mathematical & computational physics, particle and nuclear physics, physical chemistry, pure 
mathematics, mathematical physics, fluid dynamics, actuarial science, applied information economics, 
astrostatistics, biostatistics, business statistics, traffic routing in telecommunication systems (24.), cyber- 
physical security (25.), intelligent transportation systems (26.) and smart grids(27.). Consider the 
unconstrained optimization problem. 


(p); min{f (x); x E R” } (1) 


where f; R” > R continuously differentiable. The nonlinear conjugate gradient (CG) method 
usually takes the following iterative formula 


For solving (1), where x, is the current iterate point, k > O is a step length, and dg is a search 
direction defined by, 


ice | -Vf(x)ifk=1 


—Vf (xk) + Prdr-1if k = 2 ©) 
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Where ggthe gradient of f(x) , Pg E R is a scalar which determines the different conjugate gradient methods. 
(2) and (3). Well-known formulas for B; are called Conjugate-Descent (CD) (Fletcher 1997) [1], Fletcher- 
Reeves (FR) (Fletcher and Reeves 1964) [2], Hestenses.Stiefel (HS) (Hestenes and Stiefel 1952) [3], 
Liu.Storrey (LS) (Liu and Storey 1992) [4], and Polak. Ribière.Polyak (PRP) (Polak and Ribière 1969; 
Polyak (1969) [5], and some modified formulas (Dai 2002 [6]; Dai and Yuan 2001 [7], 1995 [8], Qi et al. 
1996 [9], Wei et al. 2006b [10]). The convergence behavior of the different conjugate gradient methods with 
some line search conditions (Armijo 1966 [11], Al-baali 1985 [12], Dai 2001 [13], Dai et al. 1999 [14], Dai 
and Yuan 1996, Grippo et al. 1986 [15], Grippo and Lucid 1997 [16], Liu and Han 1995 [17]. 
The well-known formulas for pẹ are, 


co — NFR IF 

Be = Toroi 4) 

BER — VEC VSCD- Ek) 6) 
K aT Vf Cr) -Vf k-11) 
Hs _ Vf æa)! (Vfl) -Vf (xK-1)) 

Be = -Oeo aD (6) 
PRP _ fOD VX- Xk-1)) 

Pe WF CRI? I] ) 
DY _ VE (x) VF (xx) 
Ko WF OR )-VE Op T de [7] (8) 

T 

Bis — WO Vf k) p4 

i dk-1 VF (xk-1) I“ (2) 
T Vfl 

pWYL — ae [Fea eV? [10] (10) 

Vf(xk-1)! (VE (K-12) 


In order to find the step length (æg), we use Strong Wolf Powell (SWP) line search, 
f (Xp Xp dk) — f (Xk) < Say)VS (xk) dy (11) 


IVf (xg +X, d,)?.d,| < —oVf(x,)" dg (12) 


Where (0 <5 < Ż) and (0 < ø < 1) 


From a bibliography point of view, the Nonlinear Conjugate Gradient methods can be improved by 
using novel techniques proposed in (20.), (21.), (22.), (23)". 

In this paper, we will present the new formula in section 2. In addition, the sufficient descent 
condition and the global convergence of the new method under the inexact line search (11) and (12), in the 
following theorem. Will be presented in section 3. Finally, we will discuss the numerical results and 
conclusion in sections 4 and five respectively. 


2. THE NEW FORMULA 
In this section, we propose the new B, which is extention of the Bi yi [10] that we named By se ; 


wylM _ VFA VE k -Vf (xk) Pk dk-1 
Pe = IVF Œk-1)ll? 2) 
Where pg = WEN and ||. || means the Euclidean norm 


IVF R-II 


3. CONVERGENT ANALYSIS OF WYLM METHOD 
In this section, we will show the convergent properties of Bi Ia using inexact line searches. 
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3.1. Convergent Analysis Based on Inexact Line Search 

In this section, we will show the convergent analysis based on the inexact line search by means of 
strong Wolfe line search. We will also show that these CG coefficients will possess sufficient descent 
conditions and global convergence properties. Under this inexact line search (11) and (12). In the following 
theorem; we discuss the sufficient condition, 


Vf) dk < —C IVF (x, )II* ,C > 0 (14) 
Where k > 0 and C E (0,1) under SWP line search. 


3.2.1. Sufficient Descent Condition 
For the sufficient descent condition, we present the following Theorem, 
Theorem 1: If the sequences g,and d; are generated by the methods (2), (3) and (13) with step 


length x, determined by (10) and (11) ifo € (0, 5, then the sufficient descent condition holds. 
Proof: We use proof by induction from (3). We know that for k = 0 it is hold. Suppose that it is: 


Vf (xk) dy = VF l +B Vf xp)" dk- 


Divide ||(Vf (xx) || indicated that; 


























Vi (xp) dk eee wylM Vi)! dk-1 
WOVE? PR IVF ERI? 
DA -0V f (xk-1)" dy_1 ( = Vf (xp) dk-1 ) (15) 
Vf (xK-1) Vf (xK-1) IVE Ce MVE EŒk-1)ll 
Using (12), we have, 
VE Œp)" dk -0V f (xk) dk-1 ( VE OK) dk-1 ) 
See I EI tae) EE Bisel aed leat | E eee a ot S 
IVF RIF 7 2 IVF Ell IVF Co INIVE Œk- ue 
Vi (xn) dk > 14 OVS (Xp-1) dp-1 ( _ Vi)! dk- ) (17) 
IVF RII? IVF RII? IVF EDIF Ek- 
And applying the Cauchy-Schwartz we get, 
Vf(xk)Tdpg-1 

ee 
0s IVE DIIVA Ek- T Ue) 
This implies that, 

OVS (Xp_1) dp_1 Vi (xz) dk OVE (Xp_-1)  dp_1 

= Ne ee | ee) ee 

oe vf? =~ WV FI? nae IVF RII? (19) 
By repeating this process and the fact Vf(x,)’d, = -IVf (x)|? Owe have, 
_yk-1 i cO de c k-1 i 

D6 (20) = CVF Ce, IN? — 2 T ae (20) (20) 
Since ) (20) =<) 25a) = — 
As shown in (19) Can be written as: 
2.08 Vf dk 1 

1-20 ~ ||(Vf(xx)II2 oe? 1-20 (21) 


By making the restriction o € (0, 5) we have Vf(x,)'d, <0so by induction, Vk € 
N,Vf(x,)' dç < 0 holds. 
Now, we prove the sufficient descent property of dg ifø E (0, =) 
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Set À = 2 — —— then 0 <A < 1, and (18) turns out to be 


Vf (xz) dk 
A 2 IVF (xxl? A oe 


Thus we obtain Vf (xp)! dg < —y|lVf(Cx,)Il? Or y =—2 + — where A E (0, 1) 
The proof is completed. 


3.2.2. Global Convergent Analysis 

The following assumption is needed in order to proceed with the proof of global 
convergence properties. 

Assumption 1 

(i) The function f is bounded below on the level set R” and is continuous and differentiable in 
neighbourhood N of the level set Q ={x € R”; f(x) < f(%,)} at the initial point x, 

(11) The gradient g(x) is Lipschitz continuous in N, so a constant LÈ 0 exists, such that that 


lg) — gy) Il < Lilx — yll : For all x,y € N 
Theorem2: Suppose Assumption 1 is true, consider any CG method of form (2) and (3), where «, 
satisfied. SWP line search and the sufficient descent condition holds, then lim IVF Cx) || = 0. 


Proof 
Subtracting Vf (x) dx from both sides of (12) and using the Lipschitz condition we have 


~(1 — o VF (xn) dy < (VE (x) — VF (xp-1)) dy < L <p IIdgl ? (23) 
Therefore, 


(1-0) VE (rR) de 
L |ldxll ? 


<K;, (24) 


With (11) we obtain: 


1-0) |V Gq) T arl 
f xx) — f (Xk +%;, dk) = —8 Xp Vf (xk) dg = —ô Samo l (25) 
Moreover, from the hypothesis (1), we have that {f (xg )} is a decreasing sequence and has a limit in, 
which shows that lim f(Xk+1) < +00 and after (25) we have, 


+o > f0) — lim fC) = If O- feal = SSP ye (26) 


T 2 
Then yee < +00 
k 


Hence, lim IVf (x;,)|| = 0 Then proof is completed. 


4. NUMERICAL RESULTS AND DISCUSSIONS 

In this section, we present the results of our proposed method WYLM on comparison with CG 
methods FR, DY, WYL, and PRP. We will use some of test problems considered in Andrei [18] to analyze 
the efficient of By’’’” We considered €= 1076; ø = 0.1; 6 = 0.01 and the gradient value as stopping 
criteria. The tolerance E= 10~° is selected for all algorithms to investigate the rapidity of iteration of these 
algorithms towards the optimal solution, We used Matlab R2010 the performance results are shown in 
Figures 1-6. We use the following algorithm, 


Algorithm 1: B, “’™ method 


Step 1: Choose an initial point xọ E R” setk = 1.€ = 10~° set dọ = —gy = -V f (Xo) 
Step 2: Compute B; based on (13). (15), (7), (8), or (10) 
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Step 3: Compute d, based on (3); if |V f (xx)|| = 0, then stop. 

Step 4: Compute step length a, by one line search technique, let Xk} = Xg +X, dk 

Step 5: Updating new point based on. (2). 

Step 6: Convergent test and stopping criteria, if (x) < f(x,_1 ) and |V f (xo |l < €, then stop, otherwise 


go to Step 1 withk =k +15 








Table 1.Test Problems used with the Strong Wolf Condition €= 107° ; ø = 0.1; 6 = 0.01 


No Functions N Initial points 

1 Booth 2 (10, 10),(14,14),(40,40) 
2 Rosenbrock 2 (5,5),(25,25),(-12,-12) 
3 Goldstein-price 2 (2,2),(-11,-11),(-13,-13) 
4 Extended Powell 4 (7,7,7,7) (15,15,15,15) 
5 Tridiagonal 1 2 (2;2), (17,17),(100;100) 
6 Strait 2 (100,100), (50;50) 

7 Generalized Quartic 2 (10,10),(200,200),(17,17) 


Figures 1-3 list the performance of the above methods relative to iterations number, the number of 
gradient evaluations and the CPU time, respectively. 





Figure 1. Performance based en the number of Figure 2. Performance based on the number of 
iterations gradient evaluations 


Benoa Fas 





Figure 3. Performance based CPU time 


Remark 1: Figures 1-3, shows that “WYLM” method has best performance since it solves about 
99% of the test problems successfully 
Example: Extended Rosenbrock function, 


f(x,y) = (1 — x)? + 100(y — x?)? , x optimal = (1,1). 
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= (-2 +2x — 400xy + __) 
am ( 200y — 200x2y 


Table 2. Extended Rosenbrock function f (x, y) = (1 — x)? + 100(y — x7), x optimal = (1, 1) Numerical 
results for BK , WY L, DY, FR and PRP in terms of number iterations (NI) and CPU time 


Initial point WYLM WYL PRP FR DY 
NI/CPU NI/CPU NI/CPU NI/CPU NI/CPU 
(1000, 1000) 290/0/8948 Failed 8377/42.896843 Failed Failed 
(30, 30) 119/0.29190 1511/ 2.613863 8377/ 42.896843 3944/ 16.488 Failed 
(-1,-1000) 316/1.2419 2324/ 7.481274 Failed Failed Failed 
(-1,3) 88/1.993644 14532/ 75.441580 466/ 2.602825 420/ 2.6787 170/ 0.21942 
(0, 1) 124/0.425383 20000/ 118.46089 510/ 3.010145 88.0.215204 100/ 0.376978 
(1,4) 224/0.809022 2000/113.847652 456/2.338696 130/ 2.086794 100 /0.376978 
(100, 100) 210/ 0.607405 Failed Failed 5840/ 25.106 Failed 
(-1, 7) 201/ 0.754304 20000/1 13.847652 527/ 6.118101 130 / 2.086794 219 / 0.442945 


Remark 2: in table 2, the WYLM method was successful in all attempts to achieve the optimal 
solution, while the other methods failed. 


Table 3. Summary of Results 


Méthod Ranking The success rate 
WYLM 1 100% 

PRP 2 75% 

FR 3 15% 

WYL 4 715% 

DY 5 50% 


Remark 3: Table 3, shows that “WYLM” has best performance since it solves about 100% of the 
test problems successfully. 


Table 4. Rosenbrock Function, Initial Point [-3 100] 


WYLM WYL DY PRP FR 
NI 309 Failed 3505 Failed 2456 
CPU(s) 1,0048 Failed 13,2389 Failed 8,5297 


X optimal (1,1) (1,1) (1,1) 


Figures 4 and 5, list the comparison of WYLM method and DY, WYL, PRP, FR methods 
x0= [-3 100]. 





Figure 4. Performance profiles based on the number of Figure 5. Performance profiles based on the 
function evaluations number of gradient evaluations 
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Remark 4: From the Figures 4-5, The WYLM method performs better than other methods by 
selecting a starting point with the Resenbrock function f (x, y) = (1 — x)? + 100 (y — x*)? 
And she is best performance in terms of values gradients and functions and the number of iterations. 


5. CONCLUSION 

We present a new CG formula B¥Y™ that similar to WYL method and the global convergence 
properties are presented with several line searches. Results showed that the new coefficient satisfies the 
sufficient descent conditions and converge globally under strong Wolfe-Powell line search.it 1s proved that 


the algorithm is practical and effective to be used. Our numerical results have shown that our new method 
has the best performance compared to the FR, PRP, DY and WYL Methods 
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